A novel square-root Nyquist filter design with prescribed ISI energy

In this paper, a novel method is introduced to design Nyquist filters with prescribed inter symbol interference (ISI) energy. The computation is based on the iterative weighted least-squares optimization with a quadratic constraint. The weighted stop band and the tail energies of the Nyquist filter are minimized iteratively while the quadratic constraint prescribing the ISI energy is enforced. The Lagrange multiplier method and generalized singular value decomposition are utilized to solve the problem. By changing the weighting coe cient, one can make a trade-o↵ between the stop band energy and the tail energy (robustness to timing jitter) while the ISI energy is predetermined. The designed Nyquist filter, can be factorized to a minimum phase and a maximum phase FIR pulse shaping square-root Nyquist filters for the transmitter and receiver with a simple modification.